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ABSTRACT 

The two techniques for improving the quality of 
programed instruction, empirical revision^ and the presentation of 
objectives before instruction were tested by this experiment in order 
to obtain a systematic replication of former experimental results • 
This study was designed to investigate: whether the presentation of 
the specific objectives technique consistently resulted in improved 
student performance; whether empirical revision resulted in improved 
student performance across subject matter areas and student 
populations; and whether a combination of both techniques would have 
a greater effect than either one alone. Using a 2x2 research design 
employing placebo objectives and using student perfi^rmance as the 
dependent variable, the study showed that the combination 
revision/objective treatment was statistically superior to either 
technique alone. The experiment was conducted on Officer Candidate 
Personnel at the United States Army Infantry School, (MC) 
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The development of high quality instructional programs has been 
a persistent problem for instructional technology.. High quality is 
dependent upon student performance following instruction. Two 
techniques for improving the quality of programmed instruction have 
demonstra,^ed their effectiveness in experimental studies. 

The first technique (Silberman, et _a_l_. , 1964) is the revision 
of the instructional program by empirical tryout with revisions 
based on student difficulties and comments. This procedure is 
followed-by subsequent trials until the satisfactory level of student 
performance is attained. 

The second technique .is the presentation of specific objectives 

before receiving instruction (Mager and McCann, 1961). This 

technique is far less costly than the first and may be applied with 
far less trouble. 

The presentation of specific objectives to students prior to 
learning is a relatively simple modification to a program. It 
requires only that the objective be stated in a way that will contri- 
bute to the desired change when presented to the students. Empirical 
revision on the other hand, is a complex and involved process requir- 
ing the extended time of a reviser and the time of trial subjects plus 
the costs of making the necessary revisions. Additionally, the process 
of revising a program on the basis of student performance often appears 
to be ppculiar to the individual conducting it (Markle, 1967b, p. 124). 
From the same data it is quite probable that different revisers will 
follow very different plans of program improvement. The implication 
is clear; if presenting the objectives before the program proves as 
effective as empirical revision, the cost and effort is considerably 
less and the former strategy should be employed. 

There is, of course, a key question that may be asked of the two 
major treatments cited (objectives before instruction and empirical _ 
revision): do these interventions produce reliable results? A particu- 
lar experimental treatment may not be significant in reality, but if it 
is tested repeatedly it is likely to appear significant, in a few attempts 
(Sidman, 1960, p. 45). The few attempts that result in an estimate of 
significance are the experiments that are most likely to be published, 
particularly if editors continue to be more sensitive to significant 
results than scientific rigor (Lykken, 1967, p. 159). A replication 
of the results supporting the two major treatments in this study under 
different circumstances would lend credence to their general izibility. 

If a particular experimental treatment is sufficiently powerful , 
it should give relatively comparable results when other factors in the 
environment vary. Many studies in educational research consist of short 
treatments speedily analyzed and reported (Shulman, 1970, p. 391). _ 
Experimenters should attempt systematic replication (Sidman, 1960, p. Ill 
where the findings in question hold up under varying conditions, such as 
student populations, subject matter areas, and programmers (or revisors}. 



The present study was designed to investigate the following 
questions: 1) Does the presentation of specific objectives prior 
to programmed instruction consistently result in improved -tudent 
performance across subject matter areas and student populations? 

2) Does empirical revision result in improved student performance 
across subject matter areas, student populations, and revisions? 

3) If both techniques are employed together, does one result greater 
levels of student performance than the other? 

The dependent variable was student, performance on the criterion 
test for the specific instructional sequence utilized. The independent 
variables were the type of objective given to students prior to learning 
(specific or placebo) and the revision status of the program (revised or 
unrevised). The placebo objectives were content and goal relevant infor- 
mation lacking the specificity of the other treatment. The unrevised ^ 
program was instruction that did not bring about the desired level of 
student achievement. The empirically revised program was the modification 
made to the unrevised program by one reviser. In order to compare the 
unrevised program by one reviser. In order to compare the effects of 
the major variables and to test any interaction a 2 x 2 research design 
was the basis for analysis (See Figure 1). 
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Figure 1. 2x2 Research Design 



From Figure 1 it may be seen that there were four treatment groups 
as follows: 

(C) Control —Placebo objectives and unrevised program 

(0) Objective— Specific objectives and unrevised program 

(R) Revised— Placebo objectives and revised program 

(RO) Revised/Objective— Specific objectives and revised program 



Since the '.study involved nine separate replications, the final 
analysis was made with a 2 x 2 x 9 factorial design. 

The null hypotheses for these tests were as follows: 

1. There will be no significant differences on the criterion 
test scores between the treatment groups given empirically revised 
programs (R,RO) and the treatment groups given the unrevised programs 
(C,0). 

2. There will be no significant differences on the criterion 
test scores between the treatment groups given specific objectives 
prior to the programs (0,R0) and the treatment groups given a placebo 
objective prior to instruction (C,R). 

3. There will be no significant interaction among the major 
treatments of revision, objectives, or replications. 

No hypotheses were formulated concerning the different replications 
as a main effect. Since different criterion instruments were utilized 
for different programs, it was necessary to equate student performance 
across replications. This was accomplished by converting the dependent 
variable scores to z-scores; therefore, the equated groups should not 
differ significantly from each other. The possibility still existed 
that there could be significant interactions between the replications 
and the objective and/or revision variables; therefore, the null hypo- 
theses concerning interactions of the replications was asserted. 

METHOD 

Revision . Two empirical revision sessions were conducted to revise . 
existing programs. The first session was two weeks and the second three 
weeks. The general procedures for the revision process are described , 
elsewhere (Sulzen, 1970), The final revision session (which provided 
six of the revisions) incorporated the following specific practices: 
(1) a personnel selection techniqe (Lysaught and Pierleoni, 1970) to 
obtain effective revisors, (2) an opportunity for revisors to try out 
materials on four individual students (developmental trials) and on 
four groups, and (3) replicable instruction in the form of nine selected 
lessons from a tape and slide program (Deterline, 1970). 

Programs . Programs (Table I) varied in format and length. All 
programs were in tape and slide format except for one programmed text. 
The tape and slide programs had questions inte, arsed throughout the 
instruction. These were multiple choice quest; s for eventual use in 
a. student resoonse classroom using the EDEX system. This system allows 
students to respond to a question by select: c an alternate at his desk 
on a responder. The responses are displayed iinmediately for instructor 
information and autoiriatically recorded for later analysis. The length 
of time for each program varied between half an hour and two hours (in^ 
no case did the revised program exceed the unrevised program in length). 



Criterion Tests . Table I shows the criterion "responses" for each 
program. The term "response" was used instead of item because a one- 
to-one relationship did not exist. Some items required more than one 
response. Scores reflected a weighting system in use by the Instruc- 
tional Department of the Infantry School with responsibility for that 
particular subject matter. 

Three types of studeiit respvonses were required in the criterion 
tests. These responses were selecting the proper alternative for 
multiple choice questions, matching a term or number to an appropriate 
description for matching questions, and writing a short answer for 
competion questions. The short answer was usually no more than two or 
three words. 

Subjects . The subjects in the experiment were members of the U.S. 
Army undergoing leadership training at the U.S. Army Infantry School 
Fort Benning, Georgia, during the summer of 1970. They were either 
Officer Candidates or Noncommissioned Officer Candidates. They were 
assigned to companies (classes) based upon when they arrived at the 
school. Classes were utilized for the experiment. Selection of each 
cla«;s was based upon its possessing no prior training on the subject 
matter presented and upon administrative constraints. All leadership 
candidates at the Infantry School were in a stressful situation during 
the titne period of the study. The typical morning began at 5:30 a.m. 
with riavaiHe, breakfast at 6:00 a.m., an inspection of quarters at 
7'00 a.m., and company formation to go to class at 7:30 a.m.. Each 
day consisted of 8 to 12 hours of training either in a classroom, on 
a firing range, or in a field exercise. The ages of the candidates 
were between 18 and 24. Although some candidates were older, they were 
eliminated from the study because they typically had received prior 
training on the subject matter presented. Another cause of experimental 
mortality was the assignment of the duty Charge-of-Quarters (CQ). These 
candidates were required to stay awake most of the night and still par- 
ticipate in the training the next day. 

Staff. Members of the Instructional Systems Branch at the U.S. Army 
InfantryTchool served as the experimental staff. The staff insured that 
all companies (classes) used in the experimental replications were ran- 
domly assigned to ore of the four major treatment groups. They presented 
the objectives given orior to the instruction in the prescribed manner. 
They were present during the instruction to detect the presence of extra- 
neous factors. They administered the immediate criterion test following 
the instruction, and they scored the criterion tests following the repli- 
cation. 

Setting. The U.S. Army Infantry Hall was the site for all tests and 
consi sted of a large building erected in the mid-1960s. There were several 
classrooms of two si:ies: 200-man and 50-man. The building had a thermo- 
statically controlled environment at 75 degrees. 



TABLE I 

INFORMATION ON EXPERIMENTAL REPLICATIONS 



Replication Subjects 



1 



Program 
Title 



Program 
Length 



Criterion 
Responses 



Time of 
Day 



1 

2 

3 

4 

5 

6 

7 

8 

9 



OC 

oc 

OC 
NCOC 
NCOC 
OC 
OC 
OC 
OC 



Electronic 1/2 
Warfare 

Evasionn ^ 1 
and Escape 

Supply 2 
Procedures 

Patrolling 1 ■ 

Terms 

4 

Fundamentals 2 
of Defense 

4 

Fundamentals 2 
of Defense 

3 

Supply 2 
Procedures 

Maintenance 2 
Records 

Evasion 1 
and Escape 



21 

20 

34 

44 

49 

51 

34 

35 

35 



Afternoon 

Afternoon 

Morning 

Evening 

Evening 

Morning 

Morning 

Afternoon 

Afternoon 



^OC = Officer Candidate; NCOC = Noncommissioned Officer Candidate. 

^Revised twice because of a change in objectives. 

^Revised twice because of a poor first revision. 

^Revised by different revisors; one for each target population. 

^Approximate time in hours. 

^General time of program and test administration. 



Treatments . There were four treatments: 1) the first was the 
control group (C) which received the unrevised instructional program 
and a placebo objective before the instruction; 2) the second group 
was-the objective group (0) which received the same unrevised instruc- 
tional program, but was given a specific objective prior to the program; 
3) the third group was the revision group (R) which received an empiri- 
cally revised program and a placebo objective; 4) the fourth group was 
the revision and objective group (RO) which received both the revised 
program and a specific objective. 

Procedure . A company (class) was administratively assigned to 
participate in the experiment. Individual students were randomly 
assigned to one of the four treatment groups. The two groups assigned 
to receive the unrevised instruction (C,0) went to one classroom while 
the groups assigned to receive the revised instruction (R,RO) went to 
a second classroom. Inside the classroom the group assigned the specific 
objective (0,R0) sat on one side, while the group assigned the placebo 
objective (C,R) sat on the other side. The appropriate objectives were 
placed face down on the tables. A member of the experimental staff then 
read the following statement: 

■ ■ LISTEN CAREFULLY TO THESE INSTRUCTIONS: 

}. Keep the sheet of paper on your desks face down. 

2. On this paper you will. find the. objectives for the 
instruction you are about to receive. 

3. WHEN you are told to do so, read the sheet carefully. 

It should help you to answer questions on the examination 
you will receive later. 

4. You may turn the objective sheet over now. 

5. Put your name and roster number on the top right corner. 
(Pause a few seconds) 

6. Read the objectives carefully. Try to remember them. 

The students were then allowed to read the objectives. The objec- 
tives were collected when all the students had finished. The sheets 
with objectives having both names and roster numbers were used to verify 
test results with assigned treatment. • 

The students were then presented either the unrevised program (C,0) 
or the revised program (R.RO). Immediately following the program the 
subjects were given the criterion test. The tests were scored and then 
each individual was identified by treatment group. This assignment was 
verified by checking the student name and roster number written on the 



objective sheets. The class leaders were consulted and any student 
assigned late night duties was eliminated from the experiment. Scores 
were tabulated and submitted for analysis. 



RESULTS 

. The dependent variable for each replication in the major experiment 
was statistically equated. This was accomplished by converting all the 
scores in one replication to z-scores (based on the performance of all 
subjects in the replication). Once replications with different dependent 
variables were eiquated by the z-scores, a 2 x 2 x 9 analysis of variance 
was made using the z-scores. The summary of this analysis (shown in 
Table II) indicated that the main effects of both revision and objectives 
were significant at the .01 level. Replications could not be significant 



TABLE II 



ANALYSIS OF VARIANCE SUMMARY TABLE FOR THE EFFECT OF 
PRESENTING SPECIFIC OBJECTIVES BEFORE INSTRUCTION AND OF 
EMPIRICAL REVISION ON THE CRITERION BEHAVIOR OF MILITARY STUDENTS 



Source 


SS 


df 


MS 


F 


Revision 


38.81 


1 


38.81 


47.43* 


Objectives 


5.87 


1 


5.87 


7.18* 


Repl i cations 


.24 


8 


.03 


.04 


Revision X 
Objectives 


.08 


1 


.08 


.10 


Revision X 
Repl i cations 


22.70 


8 


2.84 


3.47* 


Objectives X 
Replications 


5.54 


8 


.69 


.85 


Three Factor 
Interactions 


8.99 


8 


1.12 


1.37 


Experimental 
Error 


256.13 


313 


.82 





♦Significant at the .01 level 



since they were equated by z-score conversion for each replication. 
The only interaction that was significant was the interaction betwean 
revision and replicationSo this interaction was the reruU of tha 
differential success enjoyed by the revisors. 

The most appropriate post hoc analysis was considered to be a 
comparison of each of the four mnjor treatments (control, objectives, 
revision and revision/objectives). A Newman-Keuls method was employed 
to make the comparison which is shown in Table III. The results indicate 
that all other treatments were significantly better in terms of student 
performance than the control. Empirical revision was significantly better 



TABLE III 

NEWMAN-KEULS TESTS ON THE DIFFERENCES 
BETWEEN THE FOUR MAJOR TREATMENT MEANS 



Treatments 




Control 


Objective 


Revision 


Revision/Obj 






Means 


-.476 


-.188 


.222 


.441 




Control 


-.476 




.288* 


.698** 


.917** 




Objectives 


-488 






.419** 


.629** 




Revision 


.222 








.219 




Revision/ 
Obj 


.441 


















r=2 


r=3 


r=4 






99 (r,120) 
95 (r,120) 
99 (r,120) 

95 (r,120) 


3.70 

2.80 
.355 

.269 


4.20 
3.36 
.403 

.323 


4.50 
3.69 
.432 

.354 





*Significant at the ^05 level 
^^Significant at the .01 level 



than presenting. objectives prior to instruction, but util ization of both 
objectives prior to instruction and empirical revision was not signifi- 
cantly better than empirical revision alone. Nevertheless, it is evident 
that the revision/objective treatment group was numerically superior to 
the revision group and the results approach significance. Inspection of 



the scores for different replications reveals that the revision/objective 
group was consistently high or the highest. It is possible that a direc- 
tional hypothesis (Hi: RO>R>0>C) might hold with a different statistical 
technique (see Jonckheere, 1954) but this type of relationship was not 
predicted in this study. 

DISCUSSION 

It may be concluded that each of the null hypotheses may be rejected 
with a result significant at the .01 level. In the case of the interaction 
hypotheses, only that portion dealing with the interaction of revision and 
replications may be rejected. '"A'Newman-Keuls post hoc analysis of the four 
treatment variables indicates the following relationship: RO = R>0>C. 

The answer to the research questions are as follows: The presentation 
of objectives prior to programmed instruction was found to be a significant 
variable as measured by student performance replicated across student popu- 
lations and subject matter areas. Empirical revision was found to signifi- 
cantly improve student performance across subject matter areas, student 
populations, and revisions. When both empirical revision and the presen- 
tation of objectives are employed together, revision results in signifi- 
cantly higher student performance. This last finding must be viewed in 
light of the fact that empirical revision required 80 to 120 man-hours 
per program, as opposed to approximately one man-hour per program for 
writing the specific objectives. 

The results of this study reaffirm by replication the effectiveness 
of empirical revision and the presentation of specific objectives to learners 
prior to programmed instruction. The fact that these experimental variables 
provecJ effective under the systematic replication of this study (where the 
conditions of subject matter areas, student population, and reviser were 
varied) lends even stronger credence to their generalizability. It is 
recomisiended that these practices be continued by educational technologists 
when developing instructional materials. 

'Empirical revision is a powerful technique for improving student per- 
formance following programmed instruction, but it is dependent upon the 
effectiveness of the reviser. In the study conducted, a technique for 
selertina instructional programmers with standard psychologic^tl tests 
(Lysaught and Pierleoni, 1970) was not entirely accurate in predicting 
successful revision, but the revisers selected by this method were not 
volunteers. Both the experimenter and the Instructional Systems Branch 
were of the opinion that the selection procedure predicted those capable 
of successful revision, hut could not predict those motivated to do so. 
It is recommended that those using the Lysaught-Pierleoni selection 
technique draw programmers or revisors only from volunteers. 



Although empirical revision appears to be more effective than the 
presentation of ' objectives prior to learning, the empirical revision 
procpss is far more costly in terms of man-hours than the objectives 
technique (80 to 1 and 120 to 1). When cost is an important factor 
and the student performance is close to the desired level, it is recom- 
mended that objectives be presented to the learner prior to instruction 
instead of undertaking a complete empirical revision. When time is an 
importnat factor and the student performance is close to the desired 
level, it is recommended that objectives technique. be employed. When 
student performance is well below the desired achievement level, empir- 
ical revision is the recommende'i approach. 
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